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REMARKS 

Claims 1-28 are currently pending in the application, with claims 1, 4, 7, 22, 24 and 26 
being independent. Claims 1-28 were pending prior to the Office Action. 

The Examiner is respectfully requested to reconsider the rejections in view of the 
amendments and remarks set forth herein. Applicant respectfully requests favorable 
consideration thereof in light of the comments contained herein, and earnestly seeks timely 
allowance of the pending claims. 

Summary of the Interview 
To advance prosecution of the above-identified application, an Interview with the 
Examiner was undertaken on February 11, 2008. During the Interview, the independent claims of 
the application as presented in the Amendment submitted on December 26, 2007 were discussed, 
and it was explained how the claimed invention is different from Blank and US 5,577,179. At the 
conclusion of the Interview, the Examiner agreed that the prior art of record (Blank and US 
5,577,179) is different from the current invention; however, an agreement was not reached with 
respect to the claims. 

Claim Rejections - 35 USC §112 

The Examiner rejected claims 1-3, 1045, 21, 27, 4-6, 16-20, 23 and 28 under 35 U.S.C. § 
112, second paragraph. The Examiner stated that the term "a level of certainty" in claims 1 and 4 
is allegedly a relative term which renders the claims indefinite. 

Applicant traverses this rejection. 

The Examiner alleged that the term "a level of certainty" is not defined by the claim; that 
the specification does not provide a standard for ascertaining the requisite degree, and that one of 
ordinary skill in the art would not be reasonably apprised of the scope of the invention. 

Applicants respectfully disagree. Applicants point out that claims 1 and 4 set forth the 
subject matter that Applicants regard as their invention, and point out and distinctly define the 
metes and bounds of the claimed subject matter. Applicants respectfully submit that the 
Examiner's focus during examination for compliance with the requirement of definiteness in § 
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112, second paragraph is whether the claim meets the threshold requirements of clarity and 
precision . To do this, the Examiner needs only ensure that the claims define the invention with a 
reasonable degree of particularity and distinctness. Some latitude in the manner of expression 
and the aptness of terms should be permitted even though the claim language is not as precise as 
the Examiner might desire. See M.P.KP. § 2173.02. It is respectfully submitted that claims 1 
and 4 define the invention with a reasonable degree of particularity and distinctness. 

The essential inquiry under § 1 12, second paragraph is whether the public would be able 
to determine the claims' scope. Definiteness of claim language must be analyzed, not in a 
vacuum, but in light of: 

(A) The content of the particular application disclosure; 

(B) The teachings of the prior art; and 

(C) The claim interpretation that would be given by one possessing the ordinary level of skill in 
the pertinent art at the time the invention was made. 

The Examiner alleges that the specification does not provide a standard for ascertaining 
the requisite degree with respect to the term "a level of certainty". 

The test for definiteness under 35 U,S.C. 1 12, second paragraph, is whether 6t those 
skilled in the art would understand what is claimed when the claim is read in light of the specifi- 
cation/' Orthokinetics, Inc. v. Safety Travel Chairs, Inc., 806 F.2d 1565, 1576, 1 USPQ2d 1081, 
1088 (Fed. Cir. 1986). See M.P.KP. § 2173.02. Applicant respectfully submits that the present 
specification provides the requisite guidance regarding the scope of "a level of certainty" recited 
in claims 1 and 4. The specification provides various standards for measuring a "level of 
certainty" at page 6 lines 13-18, page 6 lines 23-29, page 6 line 30-page 7 line 5 and page 7 line 
31 -page 8 line 3. For example, page 6 lines 13-18 indicate that "the correction processing unit 20 
determines whether a boundary part is a boundary part with high certainty as a contour of a 
person or a boundary part with low certainty as a contour of a person from the boundary of the 
person area and the background area, which are used for the extraction of the person area in the 
person area extraction unit 14, or a boundary detected from an external circumference of the 
extracted person area"; and page 6 lines 23-29 indicate that "the boundary parts with low 
certainty correspond to, for example, a boundary part where a length between coordinate points 
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on the boundary is partially larger than a decided value due to unevenness of the boundary, a 
boundary part which is out of a range of a reference contour line (reference contour line of a 
person including a head, a neck, shoulders, etc.), which is collected from contours of a large 
number of people, added with a predetermined margin, or a boundary part which has a shape of 
each part of the boundary largely different from a shape of the reference contour." Applicant 
respectfully submits that one of ordinary skill in the art, upon reading the present specification, 
would understand the scope of "a level of certainty". Hence, one of ordinary skill in the art, in 
view of the prior art and the status of the art, is reasonably apprised of the scope of the invention. 

Hence, the specification provides the requisite guidance regarding a "level of certainty". 
Applicant points out that one of ordinary skill in the art, upon reading the specification, would 
understand the scope of "determining a level of certainty as to whether or not the detected 
boundary is a true contour of the person for each part of the detected boundary", as recited in 
claim 1 , The Examiner need only ensure that the claims define the invention with a reasonable 
degree of particularity and distinctness. Hence, the essential inquiry under § 1 12, second 
paragraph is whether the public would be able to determine the claims' scope. The public, upon 
reading the present specification, would understand the scope of "determining a level of 
certainty" as recited in claim 1 , for example. 

The Examiner stated that "a level of certainty" is a relative term which renders the claim 
indefinite. However, M.P.E.P. points out that the fact that claim language, including terms of 
degree, may not be precise, does not automatically render the claim indefinite under 35 U.S.C. 
112, second paragraph. Seattle Box Co., v. Industrial Crating & Packing, Inc., 731 F.2d 818, 221 
USPQ 568 (Fed. Cir. 1984). Acceptability of the claim language depends on whether one of 
ordinary skill in the art would understand what is claimed, in light of the specification . See 
M.P.EP. § 2173.05(b). As explained above, one of ordinary skill in the art, upon reading the 
present specification, would understand the scope of "a level of certainty". 

In conclusion, Applicants submit that the Examiner has not met his burden and has not 
proven that the term "a level of certainty" renders claims 1 and 4 indefinite. 
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For all of the above reasons taken alone or in combination, Applicants respectfully 
request reconsideration and withdrawal of the 35 U.S.C. § 1 12, second paragraph rejection of 
claims 1-3, 10-15, 21, 27, 4-6, 16-20, 23 and 28. 

Claim Rejections - 35 USC § 103 
The Examiner rejected claims 1-6, 10, 14, 16-19, 21, 23, 27-28 under 35 U.S.C § 103(a) 
as being unpatentable over US 5,345,313 ("Blank") in view of US 7,324,246 ("Enomoto"). The 
Examiner rejected claims 7-9, 25 and 26 under 35 U.S.C. § 103(a) as being unpatentable over US 
5,577,179 ("5,577,179") in view of Enomoto. The Examiner rejected claims 11-12 under 35 
U.S.C § 103(a) as being unpatentable over Blank in view of Enomoto and in further view of 
5,577,179. The Examiner rejected claims 13, 15, 20, 22 and 24 under 35 U.S.C § 103(a) as 
being unpatentable over Blank in view of Enomoto and in further view of US 2003/0058939 
("Lee"). 

Applicant traverses these rejections. 

Applicant respectfully submits the Examiner has failed to establish a prima facie case of 
obviousness. 

The above-identified application was filed in the USPTO on February 12, 2004, and 
claims priority under 35 U.S.C 1 19 based on the prior foreign application JP 2003-035949 filed 
in Japan on February 14, 2003. 

Applicants submit herein a verified English translation of the foreign priority application 
JP 2003-035949 to perfect the priority claim. The foreign priority application JP 2003-035949 
explicitly supports claims 1-28 of the above-identified application. 

The foreign priority application JP 2003-035949 was filed by Fuji Photo Film Co., Ltd. 
on February 14, 2003. An assignment of the above-identified application to Fuji Photo Film Co., 
Ltd. was recorded with the USPTO on 2/3/2004. Assignee of the above-identified application 
was changed on 1/30/2007 to Fujifilm Corporation. A print-out of the assignment information for 
the above-identified application is provided with this Reply. 

35 U.S.C § 103(c) states that subject matter developed by another person* which 
qualifies as prior art only under one or more of subsections (e), (f), and (g) of section 102 of 
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this title, shall not preclude patentability under this section where the subject matter and the 
claimed invention were, at the time the invention was made, owned by the same person or 
subject to an obligation of assignment to the same person, 

Enomoto was filed in the USPTO on September 24, 2002, as application number 
10/252,841. On 9/19/2002 an assignment of Enomoto to Fuji Photo Film Co., Ltd. was recorded 
in the USPTO. The assignee of Enomoto was changed on 01/30/2007 to Fujifilm Corporation. A 
print-out of the assignment information for Enomoto is provided with this Reply. 

Hence, Enomoto and the above-identified application were, at the time the present 
invention was made (February 14, 2003), owned by the same person or subject to an obligation 
of assignment to the same person (Fuji Photo Film Co., Ltd.), 

Furthermore, Enomoto does not qualify as prior art under subsections 35 U.S*C § 102(a) 
or 102(b), as explained below. 

Enomoto was published on May 8, 2003, and the foreign priority application of Enomoto, 
JP 2001-297742, was not published before May 8, 2003. Therefore, Enomoto was not published 
before the invention thereof by the applicant (February 14, 2003, the date of the foreign 
application JP 2003-035949 of the above-identified application). Therefore, Enomoto does not 
qualify as prior art under 35 U.S.C § 102(a), 

Enomoto does not qualify as prior art under 35 U.S.C. § 102(b) either, because Enomoto 
was not patented or published more than one year prior to the date of the application for patent in 
the United States for the above-identified application (February 12, 2004). 

In conclusion, Enomoto is not a prior art reference in an obviousness rejection of the 
above-identified application, based on 35 U.S.C. § 103(c). 

With respect to claim 1, the Examiner used Enomoto for allegedly teaching determining a 
level of certainty as to whether or not the detected boundary is a true contour of the person for 
each part of the detected boundary* Enomoto, however, is not a prior art reference for above- 
identified application, as explained above. Blank does not disclose or suggest determining a level 
of certainty as to whether or not the detected boundary is a true contour of the person for each 
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part of the detected boundary. Therefore, the prior art of record does not teach or suggest all 
elements of claim 1. 

The prior art of record also does not teach or suggest all elements of claim 4 at least 
based on reasoning similar to that set forth above. 

With respect to claim 7, the Examiner used Enomoto for allegedly teaching identifying, 
in the detected boundary, a boundary part representing a contour of the person with low 
certainty. Enomoto, however, is not a prior art reference for above-identified application, as 
explained above. Furthermore, 5,577,179 does not disclose or suggest identifying, in the detected 
boundary, a boundary part representing a contour of the person with low certainty. On page 10 of 
the Office Action the Examiner alleged that, at col. 16 lines 48-50, 5,577,179 discloses using 
edge pixels as a boundary part representing a contour of the person with low certainty* Applicant 
points out that no such tiling is disclosed by 5,577,179. At col. 16 lines 48-50, 5,577,179 states 
that 4t the computer 130 selects the three pixels just outside the edge of the object and uses them 
as edge pixels in performing a blend operation." The "three pixels just outside the edge of the 
object" are not part of the detected boundary . Hence, they do not represent a boundary part, 
because they are not part of the boundary (they are, in fact, part of the background). The 
Examiner also alleged that "background layer directly below the current object layer" mentioned 
at col 16 lines 50-53 is a low certainty boundary part. However, coL 16 lines 50-53 states that 
"the blend operation is performed by the computer 130 in steps 428, 430, and 432 wherein the 
aforementioned three pixels are blended to the background layer directly below the current 
object layer." Neither the "aforementioned three pixels" nor the "background layer directly 
below the current object layer" are part of the detected boundary. The three pixels are outside the 
edge of the object, and the background layer directly below the current object layer is part of the 
background, not part of a detected boundary. Therefore, the prior art of record does not teach or 
suggest all elements of claim 7. 

With respect to claim 26, the Examiner used Enomoto for allegedly teaching identifying, 
in the detected boundary, a boundary part representing a contour of the person with low certainty 
(page 12 of Office Action). Enomoto, however, is not a prior art reference for above-identified 
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application. Hence, the prior art of record does not teach or suggest all elements of claim 26 at 
least based on reasoning similar to that set forth above with respect to claim 7. 

With respect to independent claims 22 and 24, the Examiner used the explanation of 
claims 1 and 15 (page 16 of Office Action), where he used Enomoto for allegedly teaching a 
level of certainty. Enomoto, however, is not a prior art reference for the above-identified 
application. Hence, the prior art of record does not teach or suggest all elements of claims 22 and 
24. 

For all of the above reasons, taken alone or in combination, Applicants respectfully 
request reconsideration and withdrawal of the 35 U.S.C, § 103(a) rejections of claims 1, 4, 7, 22, 
24 and 26. Claims 2-3, 10-15, 21 and 27 depend from claim 1 and are allowable at least by virtue 
of their dependency. Claims 5-6, 16-20, 23 and 28 depend from claim 4 and are allowable at 
least by virtue of their dependency. Claims 8-9 and 25 depend from claim 7 and are allowable at 
least by virtue of their dependency. 
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Conclusion 



In view of the above amendments and remarks, this application appears to be in condition 
for allowance and the Examiner is, therefore, requested to reexamine the application and pass the 
claims to issue. 

Should there be any outstanding matters that need to be resolved in the present 
application, the Examiner is respectfully requested to contact Corina E. Tanasa, Limited 
Recognition No. L0292 under 37 CFR §1 L9(b), at telephone number (703) 208-4003, located in 
the Washington, DC area, to conduct an interview in an effort to expedite prosecution in 
connection with the present application. 

If necessary, the Commissioner is hereby authorized in this, concurrent, and future 
replies, to charge payment or credit any overpayment to Deposit Account No. 02-2448 for any 
additional fees required under 37 C.F.R. §§ 1.16 or 1.17; particularly, extension of time fees. 
Dated: June 9. 2008 Respectfully submitted, - ; s 
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[Title of the Document] Specification 

[Title of the Invention] Portrait Image Processing Method and Apparatus 
[Scope of the Patent Claims] 
[Claim 1 ] A portrait image processing method comprising the steps of: 
5 extracting a portrait image from an original image including a person and a 

background; 

compositing the extracted portrait image and a background image prepared in 
advance to create a composite image; 

detecting a boundary of the person and the background from the original 

10 image; 

judging whether or not the detected boundary is a true contour of the person 
for each part of the boundary; and 

applying correction processing for concealing a boundary part, which is 
judged not to be a true contour of the person, to the boundary of the person and the 
background in the created composite image. 

[Claim 2] The portrait image processing method according to claim 1, 

wherein said correction processing is image processing for overwriting 
another image on the boundary part which is judged not to be the true contour of the 
person. 

[Claim 3] The portrait image processing method according to either claim 1 or 2, 
15 wherein said correction processing is image processing for shifting the 

portrait image such that the boundary part, which is judged not to be the true contour 
of the person, is outside a frame of the composite image. 

[Claim 4] A portrait image processing apparatus comprising: 

a portrait image extracting device which extracts a portrait image from an 
20 original image including a person and a background; 

a background image recording device which stores a background image to be 
a background of a portrait image; 

an image compositing device which composites the extracted portrait image 
and the background image read out from said background image recording device to 
25 create a composite image; 

a boundary detecting device which detects a boundary of the person and the 
background from the original image; 

a judging device which judges whether or not the detected boundary is a true 
contour of the person for each part of the boundary; and 
30 an image correcting device which applies correction processing for 

concealing a boundary part, which is judged not to be a true contour of the person, to 
the boundary of the person and the background in the created composite image. 
[Claim 5] The portrait image processing apparatus according to claim 4, 

wherein said image correcting device performs image processing for 
35 overwriting another image on the boundary part which is judged not to be the true 
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contour of the person. 

[Claim 6] The portrait image processing apparatus according to either claim 4 or 

5, 

wherein said image correcting device performs image processing for shifting 
5 the portrait image such that the boundary part, which is judged not to be the true 
contour of the person, is outside a frame of the composite image. 
[Detailed Description of the Invention] 
[0001] 
[Field of the Invention] 
10 The present invention relates to a portrait image processing method and a portrait 
image processing apparatus, and in particular, to a technique for extracting a portrait 
image from an original image and compositing the extracted portrait image and a 
background image which is prepared in advance. 
[0002] 
[Conventional Art] 

Conventionally, as a method of extracting a portrait image from an original image, 
there is known a method of photographing a person with a blue screen as a 
background (blue back) to acquire an original image and making use of a difference 
of colors (chroma) to extract a portrait image from the original image and insert the 
portrait image in another image (chroma key). 
[0003] 

In addition, Patent Document 1 discloses a technique for dividing a photographed 
15 image into a plurality of blocks arranged as a matrix and sorting the blocks into a 
background block with small movement and an object block with large movement 
(block including a portrait image) for each block according to a magnitude of 
movement among frames. 
[0004] 
[Patent Document l] 
Japanese Patent Application Laid-Open No. 10-13799 
[0005] 

[Problem to be Solved by the Invention] 
However, there is a problem in that, although it is easy to extract a portrait image 
20 in the case of an original image which is photographed in a simple background such 
as the blue back, a portrait image cannot be extracted from an original image, which 
is photographed in an intricate background, satisfactorily. 
[0006] 

On the other hand, a method of sorting a portrait image from a background image 
disclosed in Patent Document 1 has a problem in that, since a magnitude of 
25 movement among frames for each block of a photographed image is utilized, the 
portrait image (a block including a person) and the background image cannot be 
sorted in the case of a still image or in the case in which the person does not move. 
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[0007] 

The present invention has been achieved in view of such circumstances, and it is an 
object of the present invention to provide a portrait image processing method and a 
portrait image processing apparatus which, even in the case in which a portrait image 
is not extracted accurately from an original image, can form an image, which is 
5 obtained by compositing the extracted portrait image and the background image, as a 
natural image. 
[0008] 

[Means for Solving Problem] 
In order to attain the above-described object, a portrait image processing method 
according to claim 1 of the present invention is characterized by comprising the steps 
of: extracting a portrait image from an original image including a person and a 
10 background; compositing the extracted portrait image and a background image 
prepared in advance to create a composite image; detecting a boundary of the person 
and the background from the original image; judging whether or not the detected 
boundary is a true contour of the person for each part of the boundary; and applying 
correction processing for concealing a boundary part, which is judged not to be a true 
15 contour of the person, to the boundary of the person and the background in the 
created composite image. 
[0009] 

Even in the case in which the portrait image is not extracted accurately from the 
original image (i.e., the boundary of the person and the background detected from the 
original image does not coincide with the true contour), all parts of the boundary are 
20 not necessarily inaccurate but include boundary parts with high certainty as a contour 
of a person and boundary parts with low certainty as a contour of a person. In claim 
1, in compositing the extracted portrait image and the background image, the image 
processing for concealing a boundary part is applied to the boundary parts with low 
certainty as a contour of a person such that a natural composite image is obtained. 
[0010] 

25 As recited in claim 2, in the portrait image processing method of claim 1, the 
correction processing may be image processing for overwriting another image on the 
boundary part which is judged not to be the true contour of the person. 
[0011] 

As recited in claim 3, in the portrait image processing method of either claim 1 or 2, 
the correction processing may be image processing for shifting the portrait image 
30 such that the boundary part, which is judged not to be the true contour of the person, 
is outside a frame of the composite image. 
[0012] 

A portrait image processing apparatus according to claim 4 of the present invention 
is characterized by comprising: a portrait image extracting device which extracts a 
portrait image from an original image including a person and a background; a 
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background image recording device which stores a background image to be a 
background of a portrait image; an image compositing device which composites the 
extracted portrait image and the background image read out from the background 
image recording device to create a composite image; a boundary detecting device 

5 which detects a boundary of the person and the background from the original image; 
a judging device which judges whether or not the detected boundary is a true contour 
of the person for each part of the boundary; and an image correcting device which 
applies correction processing for concealing a boundary part, which is judged not to 
be a true contour of the person, to the boundary of the person and the background in 

10 the created composite image. 
[0013] 

As recited in claim 5, in the portrait image processing apparatus of claim 4, the 
image correcting device may perform image processing for overwriting another 
image on the boundary part which is judged not to be the true contour of the person. 
[0014] 

As recited in claim 6, in the portrait image processing apparatus of either claim 4 
or 5, the image correcting device may perform image processing for shifting the 
portrait image such that the boundary part, which is judged not to be the true contour 
of the person, is outside a frame of the composite image. 
[0015] 

[Detailed Description of the Preferred Embodiments] 
A preferred embodiment of a portrait image processing method and a portrait 
image processing apparatus in accordance with the present invention will be 
hereinafter described in detail with reference to the accompanying drawings. 
[0016] 

Fig. 1 is a functional block diagram showing a main part of the portrait image 
processing apparatus in accordance with the present invention. 
[0017] 

As shown in the figure, a portrait image processing apparatus 10 can be constituted 
by a personal computer or the like, and includes an image data input unit 12, a person 
area extraction unit 14, a compositing processing unit 16, a background image 
storage unit 18, a correction processing unit 20, a correction image storage unit 22, 
and an image data output unit 24. 
[0018] 

Original image data photographed by a digital still camera (hereinafter referred to 
as DSC) or the like is inputted to the portrait image processing apparatus 10 via the 
image data input unit 12. Note that this original image data is portrait image data 
which is obtained by photographing a person in an arbitrary background as shown in 
Fig. 2(A). As the image data input unit 12, a USB interface, an infrared ray 
communication (IrDA) interface, an Ethernet interface, and a wireless 
communication interface may be used other than a media interface for DSC media. 
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A user chooses appropriately, according to a medium in which the original image data 
is stored or a recording format of the original image data, which interface is applied 
to the image data input unit 12. 
[0019] 

The original image data inputted to the portrait image processing apparatus 10 is 
outputted to the person area extraction unit 14, where a person area is distinguished 
from a background area and extracted (see Fig. 2(B)). 
[0020] 

As a method of extracting the person area from the original image, for example, 
characteristics extracting processing is performed for extracting facial parts such as 
eyes, a nose, a mouth in the original image. In the characteristics extracting 
processing, wavelet transformation is performed with respect to the original image to 
extract and quantize wavelet coefficients of appropriate positions and frequencies. 
The wavelet coefficients are subjected to matching processing to be matched with 
facial parts dictionary data which is prepared by performing the same characteristics 
extracting processing for a large number of sample images in advance, and facial 
parts are extracted. 
[0021] 

A part from which the facial parts are extracted is judged to be a face, to which 
division processing according to colors and textures is applied. This is processing 
for collectively dividing areas of similar colors and textures, and for example, a skin 
color area including coordinates of eyes is set as a face area and an area of black or 
brown slightly above the coordinates of eyes is set as a hair area to extract a person 
area. In addition, matching processing for matching person area dictionary data, 
which indicates an average positional relation between a position of eyes and a 
boundary of a person and a background, and a boundary of a person area and a 
background area obtained from the original image is performed to find a boundary of 
a person area and a background area. 
[0022] 

In addition, there are other methods such as a method of applying filter processing 
for extracting a boundary of a person and a background from a high-frequency 
component in an original image to extract a person area from the original image, a 
method of extracting a skin color in an original image, sequentially applying area 
extension to connected areas, which seem to belong to an identical area, from a 
certain point of the skin color area, extracting a face area depending upon whether or 
not an area extracted in this way is a shape of a face, and extracting a hair area above 
the face area, a neck and chest area below the face area, and the like, to thereby 
extract a person area. Note that there are various methods are possible as a method 
of extracting a person area from an original image, which is not limited to the 
above-described methods. 
[0023] 
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Image data of the person area extracted by the person area extraction unit 14 is 
outputted to the compositing processing unit 16. The compositing processing unit 
16 composites portrait image data, which is inputted from the person area extraction 
unit 14, and background image data, which is read out from the background image 
storage unit 18, and outputs composite image data of the portrait image data and the 
background image data to the correction processing unit 20. 
[0024] 

Fig. 2(C) shows a composite image obtained by compositing a portrait image with 
a background image. Note that, as the background image, a desired background 
image to be composited with the portrait image may be selected out of a plurality of 
background images stored in the background image storage unit 18 in advance or a 
background image to be composited with the portrait image may be inputted 
separately. A method of acquiring the background image is not limited to this 
embodiment. 
[0025] 

First, the correction processing unit 20 determines whether a boundary part is a 
boundary part with high certainty as a contour of a person or a boundary part with 
low certainty as a contour of a person from the boundary of the person area and the 
background area, which are used for the extraction of the person area in the person 
area extraction unit 14, or a boundary detected from an external circumference of the 
extracted person area. 
[0026] 

Fig. 3 is a diagram showing boundary parts with high certainty as a contour of a 
person and boundary parts with low certainty as a contour of a person in a boundary 
of a person area and a background area. Encircled boundary parts indicate parts 
which are determined to have low certainty as a contour of a person. 
[0027] 

The boundary parts with low certainty correspond to, for example, a boundary part 
where a length between coordinate points on the boundary is partially larger than a 
decided value due to unevenness of the boundary, a boundary part which is out of a 
range of a reference contour line (reference contour line of a person including a head, 
a neck, shoulders, etc.), which is collected from contours of a large number of people, 
added with a predetermined margin, or a boundary part which has a shape of each 
part of the boundary largely different from a shape of the reference contour. 
[0028] 

Note that, in the case in which a person is photographed with a screen or the like 
having a single color and a uniform density as a background, a boundary of the 
person and the background detected from an original image coincides with a true 
contour of the person in most cases. However, in the case in which a person is 
photographed in a background with an intricate pattern or color, a boundary of the 
person and the background cannot be detected accurately from an original image, and 
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the boundary of the person and the background includes parts which do not coincide 
with a true contour of the person. 
[0029] 

When the boundary part with low certainty as a contour of a person is detected as 
described above, the correction processing unit 20 performs image processing for 
concealing the boundary part. In other words, the correction processing unit 20 
reads out an appropriate correction image from the correction image storage unit 22 
and overwrites this correction image on the boundary part with low certainty as a 
contour of a person. 
[0030] 

Fig. 2(D) shows a composite image after correction in which the correction image 
is overwritten on the boundary part with low certainty by the correction processing 
unit 20. In the embodiment shown in Fig. 2(D), an image of leaves is used as a 
correction image. However, it is preferable to select an appropriate correction 
image, which does not have a sense of incongruity with respect to the background 
image, from the correction image storage unit 22. In addition, the overwriting 
processing may be performed with a hat as a correction image in the case in which 
the boundary part with low certainty is in the parietal area, or a shawl as a correction 
image in the case in which the boundary part with low certainty is on the shoulders. 
[0031] 

The composite image data corrected by the correction processing unit 20 is 
outputted as image data, in which the background is composited, via the image data 
output unit 24. Examples of a form of outputting the image data include a form of 
outputting the image data to a monitoring device or a printer, and a form of recording 
the image data in a file format in an external recording medium such as a PC card or 
a CD-ROM or a hard disk incorporated in the portrait image processing apparatus 10. 
In addition, a form of transferring the image data to other apparatuses via 
communication means is also possible. Communication may be performed by wire 
or by radio. 
[0032] 

Figs. 4 are diagrams showing another embodiment of the correction processing by 
the correction processing unit 20. 
[0033] 

Fig. 4(A) is a diagram showing boundary parts with high certainty as a contour of a 
person and boundary parts with low certainty as a contour of a person in a boundary 
of a person area and a background area. Encircled boundary parts indicate parts 
which are determined to have low certainty. As shown in the figure, the boundary 
parts with low certainty concentrate in a left part A of the boundary. 
[0034] 

In the case in which the boundary parts with low certainty concentrate only in a 
part of the boundary of the person area and the background area as shown in Fig. 
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4(A), the correction processing unit 20 shifts a portrait image such that the boundary 
part with low certainty (left part Al) is out of a composite image. 
[0035] 

Note that the correction processing unit 20 may be adapted to perform correction 
processing which is a combination of the overwriting processing according to the 
correction image shown in Fig. 2(D) and the shift of the portrait image shown in Fig. 
4(B). 

[0036] 

The portrait image processing apparatus 10 can be realized by a personal computer. 
However, the portrait image processing apparatus 10 is not limited to this but may be 
realized by a service server or the like for image processing on a network. 
[0037] 

Fig. 5 is a diagram of a network system to which the portrait image processing 
method in accordance with the present invention is applied. 
[0038] 

In Fig. 5, reference numeral 30 designates a cellular phone with camera which is 
connectable to a network 40 such as the Internet, and 50 designates a computer (PC) 
of a user which is connectable to the network 40. Note that a DSC 52 is connected 
to the PC 50 via an interface such as a USB such that the PC 50 can capture an image 
from the DSC 52. 
[0039] 

In addition, a service server 60, which performs the same image processing as the 
image processing in the portrait image processing apparatus 10, a print server 70 
which prints to output a composite image processed in the service server 60, or the 
like are connected to the network 40. 
[0040] 

In the case in which a user uses a compositing processing service of a background 
image, which is provided by the service server 60, with the cellular phone with 
camera 30 or the PC 50, the user accesses a web site of the service server 60 to 
upload an image requesting compositing processing of the background image to the 
service server 60. In addition, the service server 60 can present a list of background 
images or the like to the user and cause the user to select a background image. 
[0041] 

The service server 60 includes a server computer 62, which has the same function 
as the portrait image processing apparatus 10 shown in Fig. 1 and a communication 
function, and a large capacity storage 64, which stores the image uploaded from the 
user and user information such as a user ID and a mail address. Upon receiving a 
request for compositing processing for an original image uploaded from the user with 
a background image, the service server 60 extracts a portrait image from the original 
image, performs compositing processing of this portrait image and a background 
image selected in advance, and applies correction processing to a boundary part with 
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low certainty as a contour of a person in a boundary of a person area and a 
background area. Then, the service server 60 attaches an image, in which the 
background image is composited, created in this way to an e-mail and distributes the 
image to the cellular phone with camera 30 and the PC 50 of the user or distributes a 
mail attached with a URL for image download to the cellular phone with camera 30 
and the PC 50 of the user. 
[0042] 

In addition, when an order for printing a composite image is received from the user, 
the service server 60 transfers the composite image to a print server 70. The print 
server 70 includes a server computer 72 and a print apparatus 74, and prints to output 
a composite image, in which a background is composited, with the print apparatus 74 
on the basis of the composite image received from the service server 60. Note that a 
photograph print, which is printed and outputted by the print apparatus 74, is 
delivered to a destination such as a convenience store or a photo processing shop 
designated by the user or directly delivered to a home of the user. 
[0043] 
[Effects of the Invention] 

As described above, according to the present invention, in compositing a portrait 
image extracted from an original image and an appropriate background image, even 
in the case in which the original image includes an intricate background image and 
the portrait image is not extracted accurately from the original image (i.e., a boundary 
of a person and a background detected from the original image does not coincide with 
a true contour of the person), since the image processing for concealing a boundary 
part is applied to the boundary parts with low certainty as a contour of a person, a 
natural composite image can be obtained. 

[Brief Description of the Drawings] 
[Fig.l] 

A functional block diagram showing a main part of a portrait image processing 
apparatus in accordance with the present invention 
[Figs. 2] 

Diagrams used for explaining an embodiment of a portrait image processing 
method in accordance with the present invention 
[Fig. 3] 

A diagram showing boundary parts with high certainty as a contour of a person and 
boundary parts with low certainty as a contour of a person in a boundary of a person 
area and a background area of an original image 
5 [Figs. 4] 

Diagrams showing another embodiment of correction processing in the portrait 
image processing method in accordance with the present invention 
[Fig. 5] 

A diagram of a network system to which the portrait image processing method in 
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accordance with the present invention is applied 
[Brief Description of the Reference Numbers] 
10* ••portrait image processing apparatus, 12- -image data input unit, 
14- -person area extraction unit, 16- "compositing processing unit, 
18- -background image storage unit, 20- "correction processing unit, 
22- -correction image storage unit, 24 - • -image data output unit, 
30- • -cellular phone with camera, 40- -network, 50---PC, 52- DSC, 60- -service 
server, 70- -print server 
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[Title of the Document] Abstract 
[Abstract] 

[Object] Even in the case in which the portrait image is not extracted accurately 
from the original image, forming an image, which is obtained by compositing the 
extracted portrait image and the background image, as a natural image. 

[Means for Solution] Extract a portrait image (Fig. 2(B)) from an original image 
(Fig. 2(A)) and composite the extracted portrait image and a background image 
prepared in advance to create a composite image (Fig. 2(C)). On the other hand, 
detect a boundary of a person and a background from the original image and judges 
whether or not the detected boundary is a true contour of the person for each part of 
the boundary. Then, apply correction processing (overwrite processing of a 
correction image) for concealing a boundary part, which is judged not to be the true 
contour of the person, to the boundary of the person and the background in the 
composite image. 

[Selected Drawing] Fig. 2 
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